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Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

• If NO period for reply is specified above, the maximum statutory period wilt apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

• Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )I3 Responsive to communication(s) filed on 18 July 2005 . 
2a)M This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) S Claim(s) 1-3,5-7 and 9 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) S Claim(s) 7-3,5-7 and 9 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) S Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)!3 All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Arguments and Amendments 

1. Applicant's amendments and arguments filed 18 July 2005, have been entered and made 
of record. 



Response to Amendments and Arguments 

2. Applicant's amendments and corresponding arguments for independent claims 1 and 7 
are sufficient to overcome the previous rejections thereof. Therefore, the previous rejections of 
claims 1 and 7 have been withdrawn. However, new grounds of rejection are presented below in 
view of newly discovered prior art. 

Also, with regards to claim 6, Applicant asserts that the rejection is based on hindsight • 
and involves merely picking and choosing come features while ignoring others. However, as 
explained below, the prior art does provide the teachings and motivation for one skilled in the art 
to essentially pick and choose which sensors to use on the basis of environmental conditions, and 
the utilization of certain sensors then becomes a design issue in the framework of a specific 
application. 

Claim Objections 

3. The following sections of 37 CFR § 1.75(a) and (d)(1) are the basis of the following 
objection: 



(a) The specification must conclude with a claim particularly pointing out 
and distinctly claiming the subject matter which the applicant regards as his 
invention or discovery 
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(d)(1) The claim or claims must conform to the invention as set forth in 
the remainder of the specification and the tains and phrases used in the 
claims must find clear support or antecedent basis in the description so 
that the meaning of the terms in the claims may be ascertainable by 
reference to the description. 

4. Claims 1-3, 5, and 6 are objected to under 37 CFR § 1.75(a) and (d)(1) as failing to 
particularly point out and distinctly claim the subject matter that the applicant regards as the 
invention. 

Regarding claim 1, it appears that "three-dimensional vision data" in the last 
"wherein" clause should be changed to "three-dimensional view data" in order to be consistent 
with the corresponding feature in the "integrated view data generator" clause. Appropriate 
correction is required. 

Claim Rejections - 35 USC §102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

6. Claim 9 is rejected under 35 U.S.C. 102(b) as being anticipated by U.S. Patent 6,061,068 
by Hoffman, II et al. ("Hoffman"). 

Regarding claim 9, Hoffman discloses an integrated view system (figure 2) comprising: 
at least one stereo-camera (12) installed in a vehicle for taking images of predetermined 

outside area (column 13, lines 1-6: the sensor comprises a stereo pair of different sensors whose 

image data is fused together in a common coordinate system); 
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a stereo-image recognizer (17) for processing a pair of images taken by the stereo-camera 
to recognize objects that are obstacle to the front, thus generating obstacle data (the processor 17 
utilizes the images taken by the sensor system 12 to recognize "virtual" objects that correspond 
to e.g. stored terrain data 34 in a virtual world database 23; obstacle data corresponding to such 
virtual (i.e. "symbolized" - not real images) objects is generated at block 37 - see column 3, 
lines 53-60 and column 4, lines 18-24); 

a geographical image generator (36) for generating a geographical image viewed from 
viewing points of a pilot crew (i.e. the sensor imagery is utilized to generate a common- 
coordinate transformed measured image set that represents actual geographical terrain imagery); 

an integrated view data generator (24) for generating integrated view data including 3-D 
view data based on the pair of images taken by the stereo-camera, the obstacle data including at 
least one symbolized (i.e. virtual) obstacle from the stereo-image recognizer, and the 
geographical image from the geographical image generator (the combining block 24 combines, 
or integrates, the real imagery from the sensor 12 with the generated virtual object imagery from 
the virtual database 23 and the transformed measured image ("geographical image") to generate 
integrated data that includes data pertaining to e.g. the fused real-imagery 3-D terrain features); 
and 

an integrated image display for displaying the integrated view data as visible images 
overlapping with an actual view from a cockpit (i.e. the output images are sent to a display, as 
shown in figure 2, wherein the integrated data includes the actual view). 
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Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 US.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

8. This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1 .56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 

9. Claims 1-3 and 7 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent 6,061,068 by Hoffman, II et al. ("Hoffman"). 

Regarding claims 1 and 7, Hoffman discloses an integrated view system (figure 2) 
comprising: 

at least one stereo-camera (12) installed in a vehicle for taking images of predetermined 
outside area (column 13, lines 1-6: the sensor comprises a stereo pair of different sensors whose 
image data is fused together in a common coordinate system); 

a stereo-image recognizer (17) for processing a pair of images taken by the stereo-camera 
to recognize objects that are obstacle to the front, thus generating obstacle data (the processor 17 
utilizes the images taken by the sensor system 12 to recognize "virtual" objects that correspond 
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to e.g. stored terrain data 34 in a virtual world database 23; obstacle data corresponding to such 
virtual (i.e. "symbolized" - not real images) objects is generated at block 37 - see column 3, 
lines 53-60 and column 4, lines 18-24); 

an integrated view data generator (24) for generating integrated view data including 3-D 
view data based on the pair of images taken by the stereo-camera and the obstacle data from the 
stereo-image recognizer (the combining block 24 combines, or integrates, the real imagery from 
the sensor 12 with the generated virtual object imagery from the virtual database 23 to generate 
integrated data that includes data pertaining to e.g. the fused real-imagery 3-D terrain features); 
and 

an integrated image display for displaying the integrated view data as visible images to 
crew in the vehicle (i.e. the output images are sent to a display, as shown in figure 2), 

wherein the integrated view data generator is capable of removing the 3-D vision data 
from the integrated view data by turning off the stereo-camera while retaining the obstacle data 
(column 12, lines 50-65: when the image data from the sensor system is "unreliable," the virtual 
object data is utilized in lieu of the sensor data or a combination of the sensor and virtual data, 
thereby removing the fused real- image data and retaining only virtual objects; although Hoffman 
does not expressly state that the sensors are turned off, Hoffman teaches essentially the 
equivalent thereof in that that sensors are disabled for a period of time since they are not 
providing reliable image data). 
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Regarding claim 2, Hoffman discloses adding peripheral wide-area view data to the 3-D 
view data (i.e. the sensors capture images of a wide view of terrain taken from an airplane, which 
view- view imagery is integrated as explained above for claim 1). 

Regarding claim 3, Hoffman discloses including a head mount display for overlapping 
the visible images of the integrated view data and the actual view (column 12, lines 9-30: the 
integrated data is displayed on a head-mounted display, and the integrated data includes the 
actual view as explained above for claim 1). 

10. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
6,061,068 by Hoffman, II et al. ("Hoffman") in view of [Copeland89] (J.W. Copeland, U.S. 
Patent 4,805,015). 

Regarding claim 5, Hoffman discloses utilizing two different sensors in a stereo fashion 
(see e.g. column 13, lines 1-6), but does not expressly disclose two IR cameras, as claimed. 

[Copeland89] discloses an airborne imaging system similar to that of Hoffman, whereby 
stereo images from an airplane are captured in order to aid in navigation of the airplane. In 
particular, [Copeland89]suggests that the sensor pairs (13,14), (A,A'), (B,B'), and (C,C) may be 
infrared (IR) cameras ([Copeland89] column 5, lines 49-51) - that is, 

(5.) The stereo-camera includes two infrared cameras. As with all stereo vision 
systems, the stereo sensors are separated from each other by a distance 
corresponding to a specific baseline. 
Given this, it would have been obvious to one of ordinary skill in the art, at the time of the 
Applicant's claimed invention to use IR cameras for obtaining stereo pairs consisting of IR 
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spectrum images. IR cameras are capable of detecting objects at night or when obscured by 
shadows. Therefore, the proposed modification of [Copeland89] would result in a vision system 
capable of viewing objects in the dark. 

11. Claims 6 is rejected under 35 U.S.C. § 103(a) as being unpatentable over Hoffman in 
view of [Copeland89], as applied above with respect to Claim 5, and further view of 
[SarangapaniOO] (J. Sarangapani, U.S. Patent 6,055,042) and [OwensOO] (K. Owens and L. 
Matthies, Passive Night Vision Sensor Comparison for Unmanned Ground Vehicle Stereo Vision 
Navigation, IEEE International Conference on Robotics and Automation, April 2000). 

The following is in regard to Claim 6. [Copeland89] suggests the use of several different 
types of stereo sensors (cameras). These include, for example, digital video cameras, infrared 
cameras, radar receivers, and sonar ([Copeland89] column 5, lines 49-52). However, 
[Copeland89] does not show or suggest using these sensors together, in an alternating fashion, 
which is contingent upon the "actual views". 

[SarangapaniOO] discloses a method for detecting obstacles using multiple sensors, 
comprising: 

(6.asaranga P ani ) Using multiple sensors ([SarangapaniOO] column 2, lines 5-1 1 and Fig. 8, 
step 802). As disclosed in [SarangapaniOO], these sensors can include, but 
are not limited to, radar scanners, sonar scanners, laser scanners, optical 
cameras, and infrared (IR) cameras ([SarangapaniOO] column 3, lines 28- 
30). 

(6.bsaranga P ani ) Selectively using the sensors in accordance with the actual views (cf. 
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[SarangapaniOO] Fig. 8). [SarangapaniOO] assigns a weight to each sensor 
based on observed environmental parameters ([SarangapaniOO] column 3, 
line 61 through column 4, lines 32 and column 5, line 66 through column 
6, line 17). For example, one parameter could be the amount of ambient 
light observed. In conditions of low ambient light (e.g. night, fog dust, 
etc.), for instance, low weighting factors are assigned to sensors which 
are sensitive to the amount of ambient light. On the other hand, a larger 
weight is assigned to sensors which are not affected by the amount of 
ambient light. The weights could, for example, take on values between 0 
and 1 . In this case, under low light conditions, the weight of an optical 
sensor would approach 0, whereas less sensitive sensors, like IR or radar, 
would receive a weight closer to 1. Effectively, a sensor is selected in 
accordance with the observed environmental conditions, and its known 
amenability to such conditions (as reflected by its assigned weight). 
Conditions, such as ambient light, relate to the "actual view". 
It would have been obvious to one of ordinary skill in the art, at the time of the Applicant's 
claimed invention, modify the aforementioned combination of Hoffman and [Copeland89] to use 
several types of sensor technologies. Clearly, the benefit of using several different sensor 
technologies is that it would permit the observation of the environment in varying environmental 
conditions. With respect to the systems of Hoffman and [Copeland89], the usage of multiple 
sensors would allow obstacles to be observed in stereo, modeled, and displayed to the user, even 
in unfavorable viewing conditions. Also, provided the teachings of [SarangapaniOO], it would 
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have been obvious to one of ordinary skill in the art, at the time of the Applicants claimed 
invention, to utilize these sensors selectively in accordance with the observed environmental 
conditions (actual view). Using the different sensor technologies in this manner ensures that the 
most suitable type of sensor is chosen (or, at least, that ineffectual sensors are not used), given 
the observed environmental conditions. 

As stated above, the teachings of [SarangapaniOO] are not limited to any particular type of 
sensor technology. Hoffman and [Copeland89] both suggest the usage of IR sensors, particularly 
in conditions of low ambient light. In particular, [Copeland89] suggests applying IR sensors to 
stereo vision. However, neither Hoffman, [Copeland89], nor [SarangapaniOO] suggest the usage 
of "intensifiers". 

[OwensOO] provides a comparative study of various night vision technologies employed 
in the authors' stereo-vision navigation system. [OwensOO] contemplates using several different 
types of night vision technologies. These include the four major groups of commercial-off-the- 
shelf (COTS) night vision technologies: 3-5 micron cooled Indium Antimonide (InSb), (2) 8-12 
micron cooled Mercury Cadmium Telluride (MCT) and 8-12 micron cooled GaAs Quantum 
Well Infrared Photodetectors (QWIP), (3) 8- 12 micron uncooled microbolometers and 
pyroelectric detectors and (4) third generation image intensifiers ([OwensOO] page 123, Section 
2.0: Stereo Disparity Error Analysis, paragraph 1). 

Since image intensifiers are a well-known night vision technology, and given their demonstrated 
usage in stereo-vision systems such as that of [OwensOO], it would have been obvious to one of 
ordinary skill in the art, at the time of the Applicant's claimed invention to use image intensifiers 
as one of the multiple sensor technologies in the aforesaid combination of Hoffman, 
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[Copeland89], and [SarangapaniOO]. As just stated, image intensifies are useful in night vision. 
Image intensifiers and IR sensors detect light from different portions of the electromagnetic 
spectrum. Image intensifiers are attuned to the visible and near-infrared regions of the 
electromagnetic spectrum, whereas the cooled and uncooled FLIR sensors in [OwensOO] operate 
in the thermal range of the IR spectrum. Because thermal IR sensors detect emitted thermal 
radiation, they are less susceptible than image intensifiers (which detect the visible and near-IR 
radiation reflected from an object) to obscuration due to fog, smoke, dust, etc. Thermal imagers 
can also detect objects at further distances than image intensifiers. However, thermal imagers 
generally lack the resolution of image intensifiers. Therefore, image intensifiers and IR sensors 
can be used in a complementary fashion. Image intensifiers would be preferable in clear, low- 
light situations. Thermal IR sensors, on the other hand, could be used in instances when image 
intensifiers may fail, such as in conditions of fog, smoke, dust, or when objects are located at 
distances outside the range of an image intensifier. The presence of fog, smoke, dust, etc. and the 
distance of an object could all be treated as environmental parameters in accordance with 
[SarangapaniOO]. Selection between the sensors could be effected according to the methodology 
prescribed in [SarangapaniOO]. 

Conclusion 

12. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Colin M. LaRose whose telephone number is (571) 272-7423. 
Please note that this application has been reassigned to Colin LaRose* If attempts to reach 
the examiner by telephone are unsuccessful, the examiner's acting supervisor, Bhavesh Mehta, 
can be reached on (571) 272-7453. The fax phone number for the organization where this 
application or proceeding is assigned is (571) 273-8300. Any inquiry of a general nature or 
relating to the status of this application or proceeding should be directed to the TC 2600 
Customer Service Office whose telephone number is (571) 272-2600. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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